{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear Regression Code Along"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is the reference for the video lecture on the Linear Regression Code Along. Basically what we do here is examine a dataset with Ecommerce Customer Data for a company's website and mobile app. Then we want to see if we can build a regression model that will predict the customer's yearly spend on the company's product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing to do is start a Spark Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.appName('lr_example').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.regression import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use Spark to read in the Ecommerce Customers csv file.\n",
    "data = spark.read.csv(\"Ecommerce_Customers.csv\",inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Email: string (nullable = true)\n",
      " |-- Address: string (nullable = true)\n",
      " |-- Avatar: string (nullable = true)\n",
      " |-- Avg Session Length: double (nullable = true)\n",
      " |-- Time on App: double (nullable = true)\n",
      " |-- Time on Website: double (nullable = true)\n",
      " |-- Length of Membership: double (nullable = true)\n",
      " |-- Yearly Amount Spent: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print the Schema of the DataFrame\n",
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+\n",
      "|               Email|             Address|          Avatar|Avg Session Length|       Time on App|   Time on Website|Length of Membership|Yearly Amount Spent|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+\n",
      "|mstephenson@ferna...|835 Frank Tunnel\n",
      "...|          Violet| 34.49726772511229|12.655651149166752| 39.57766801952616|   4.082620632952961|  587.9510539684005|\n",
      "|   hduke@hotmail.com|4547 Archer Commo...|       DarkGreen|31.926272026360156|11.109460728682564|37.268958868297744|    2.66403418213262|  392.2049334443264|\n",
      "|    pallen@yahoo.com|24645 Valerie Uni...|          Bisque|33.000914755642675|11.330278057777512| 37.11059744212085|   4.104543202376424| 487.54750486747207|\n",
      "|riverarebecca@gma...|1414 David Throug...|     SaddleBrown| 34.30555662975554|13.717513665142508| 36.72128267790313|  3.1201787827480914|  581.8523440352178|\n",
      "|mstephens@davidso...|14023 Rodriguez P...|MediumAquaMarine| 33.33067252364639|12.795188551078114| 37.53665330059473|   4.446308318351435|  599.4060920457634|\n",
      "|alvareznancy@luca...|645 Martha Park A...|     FloralWhite| 33.87103787934198|12.026925339755058| 34.47687762925054|   5.493507201364199|   637.102447915074|\n",
      "|katherine20@yahoo...|68388 Reyes Light...|   DarkSlateBlue| 32.02159550138701|11.366348309710526|36.683776152869605|  4.6850172465709115|  521.5721747578274|\n",
      "|  awatkins@yahoo.com|Unit 6538 Box 898...|            Aqua|32.739142938380326| 12.35195897300293|37.373358858547554|  4.4342734348999375|  549.9041461052942|\n",
      "|vchurch@walter-ma...|860 Lee Key\n",
      "West ...|          Salmon| 33.98777289568564|13.386235275676434|37.534497341555735|  3.2734335777477144|  570.2004089636195|\n",
      "|    bonnie69@lin.biz|PSC 2734, Box 525...|           Brown|31.936548618448914|11.814128294972196| 37.14516822352819|   3.202806071553459| 427.19938489532814|\n",
      "|andrew06@peterson...|26104 Alexander G...|          Tomato| 33.99257277495374|13.338975447662111| 37.22580613162114|  2.4826077705105956|  492.6060127179966|\n",
      "|ryanwerner@freema...|Unit 2413 Box 034...|          Tomato| 33.87936082480498|11.584782999535266|37.087926070983805|    3.71320920294043|  522.3374046069357|\n",
      "|   knelson@gmail.com|6705 Miller Orcha...|       RoyalBlue|29.532428967057946|10.961298400154098| 37.42021557502538|   4.046423164299585| 408.64035107262754|\n",
      "|wrightpeter@yahoo...|05302 Dunlap Ferr...|          Bisque| 33.19033404372265|12.959226091609382|36.144666700041924|  3.9185418391589986|  573.4158673313865|\n",
      "|taylormason@gmail...|7773 Powell Sprin...|        DarkBlue| 32.38797585315387|13.148725692056516| 36.61995708279922|   2.494543646659249| 470.45273330095546|\n",
      "| jstark@anderson.com|49558 Ramirez Roa...|            Peru|30.737720372628186|12.636606052000129|36.213763093698624|   3.357846842326294|  461.7807421962299|\n",
      "| wjennings@gmail.com|6362 Wilson Mount...|      PowderBlue| 32.12538689728784|11.733861690857392|  34.8940927514398|  3.1361327164897803| 457.84769594494855|\n",
      "|rebecca45@hale-ba...|8982 Burton Row\n",
      "W...|       OliveDrab| 32.33889932306719|  12.0131946940144| 38.38513659413844|   2.420806160901484|  407.7045475495441|\n",
      "|alejandro75@hotma...|64475 Andre Club ...|            Cyan|32.187812045932155|14.715387544156501|38.244114594343515|  1.5165755808319439| 452.31567548003545|\n",
      "|samuel46@love-wes...|544 Alexander Hei...|   LightSeaGreen| 32.61785606282345|13.989592555825254|37.190503800397956|   4.064548550437977|   605.061038804892|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(Email='mstephenson@fernandez.com', Address='835 Frank Tunnel\\nWrightmouth, MI 82180-9605', Avatar='Violet', Avg Session Length=34.49726772511229, Time on App=12.655651149166752, Time on Website=39.57766801952616, Length of Membership=4.082620632952961, Yearly Amount Spent=587.9510539684005)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mstephenson@fernandez.com\n",
      "835 Frank Tunnel\n",
      "Wrightmouth, MI 82180-9605\n",
      "Violet\n",
      "34.49726772511229\n",
      "12.655651149166752\n",
      "39.57766801952616\n",
      "4.082620632952961\n",
      "587.9510539684005\n"
     ]
    }
   ],
   "source": [
    "for item in data.head():\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up DataFrame for Machine Learning "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A few things we need to do before Spark can accept the data!\n",
    "# It needs to be in the form of two columns\n",
    "# (\"label\",\"features\")\n",
    "\n",
    "# Import VectorAssembler and Vectors\n",
    "from pyspark.ml.linalg import Vectors\n",
    "from pyspark.ml.feature import VectorAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Email',\n",
       " 'Address',\n",
       " 'Avatar',\n",
       " 'Avg Session Length',\n",
       " 'Time on App',\n",
       " 'Time on Website',\n",
       " 'Length of Membership',\n",
       " 'Yearly Amount Spent']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assembler = VectorAssembler(\n",
    "    inputCols=[\"Avg Session Length\", \"Time on App\", \n",
    "               \"Time on Website\",'Length of Membership'],\n",
    "    outputCol=\"features\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "output = assembler.transform(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            features|\n",
      "+--------------------+\n",
      "|[34.4972677251122...|\n",
      "|[31.9262720263601...|\n",
      "|[33.0009147556426...|\n",
      "|[34.3055566297555...|\n",
      "|[33.3306725236463...|\n",
      "|[33.8710378793419...|\n",
      "|[32.0215955013870...|\n",
      "|[32.7391429383803...|\n",
      "|[33.9877728956856...|\n",
      "|[31.9365486184489...|\n",
      "|[33.9925727749537...|\n",
      "|[33.8793608248049...|\n",
      "|[29.5324289670579...|\n",
      "|[33.1903340437226...|\n",
      "|[32.3879758531538...|\n",
      "|[30.7377203726281...|\n",
      "|[32.1253868972878...|\n",
      "|[32.3388993230671...|\n",
      "|[32.1878120459321...|\n",
      "|[32.6178560628234...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.select(\"features\").show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+--------------------+\n",
      "|               Email|             Address|          Avatar|Avg Session Length|       Time on App|   Time on Website|Length of Membership|Yearly Amount Spent|            features|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+--------------------+\n",
      "|mstephenson@ferna...|835 Frank Tunnel\n",
      "...|          Violet| 34.49726772511229|12.655651149166752| 39.57766801952616|   4.082620632952961|  587.9510539684005|[34.4972677251122...|\n",
      "|   hduke@hotmail.com|4547 Archer Commo...|       DarkGreen|31.926272026360156|11.109460728682564|37.268958868297744|    2.66403418213262|  392.2049334443264|[31.9262720263601...|\n",
      "|    pallen@yahoo.com|24645 Valerie Uni...|          Bisque|33.000914755642675|11.330278057777512| 37.11059744212085|   4.104543202376424| 487.54750486747207|[33.0009147556426...|\n",
      "|riverarebecca@gma...|1414 David Throug...|     SaddleBrown| 34.30555662975554|13.717513665142508| 36.72128267790313|  3.1201787827480914|  581.8523440352178|[34.3055566297555...|\n",
      "|mstephens@davidso...|14023 Rodriguez P...|MediumAquaMarine| 33.33067252364639|12.795188551078114| 37.53665330059473|   4.446308318351435|  599.4060920457634|[33.3306725236463...|\n",
      "|alvareznancy@luca...|645 Martha Park A...|     FloralWhite| 33.87103787934198|12.026925339755058| 34.47687762925054|   5.493507201364199|   637.102447915074|[33.8710378793419...|\n",
      "|katherine20@yahoo...|68388 Reyes Light...|   DarkSlateBlue| 32.02159550138701|11.366348309710526|36.683776152869605|  4.6850172465709115|  521.5721747578274|[32.0215955013870...|\n",
      "|  awatkins@yahoo.com|Unit 6538 Box 898...|            Aqua|32.739142938380326| 12.35195897300293|37.373358858547554|  4.4342734348999375|  549.9041461052942|[32.7391429383803...|\n",
      "|vchurch@walter-ma...|860 Lee Key\n",
      "West ...|          Salmon| 33.98777289568564|13.386235275676434|37.534497341555735|  3.2734335777477144|  570.2004089636195|[33.9877728956856...|\n",
      "|    bonnie69@lin.biz|PSC 2734, Box 525...|           Brown|31.936548618448914|11.814128294972196| 37.14516822352819|   3.202806071553459| 427.19938489532814|[31.9365486184489...|\n",
      "|andrew06@peterson...|26104 Alexander G...|          Tomato| 33.99257277495374|13.338975447662111| 37.22580613162114|  2.4826077705105956|  492.6060127179966|[33.9925727749537...|\n",
      "|ryanwerner@freema...|Unit 2413 Box 034...|          Tomato| 33.87936082480498|11.584782999535266|37.087926070983805|    3.71320920294043|  522.3374046069357|[33.8793608248049...|\n",
      "|   knelson@gmail.com|6705 Miller Orcha...|       RoyalBlue|29.532428967057946|10.961298400154098| 37.42021557502538|   4.046423164299585| 408.64035107262754|[29.5324289670579...|\n",
      "|wrightpeter@yahoo...|05302 Dunlap Ferr...|          Bisque| 33.19033404372265|12.959226091609382|36.144666700041924|  3.9185418391589986|  573.4158673313865|[33.1903340437226...|\n",
      "|taylormason@gmail...|7773 Powell Sprin...|        DarkBlue| 32.38797585315387|13.148725692056516| 36.61995708279922|   2.494543646659249| 470.45273330095546|[32.3879758531538...|\n",
      "| jstark@anderson.com|49558 Ramirez Roa...|            Peru|30.737720372628186|12.636606052000129|36.213763093698624|   3.357846842326294|  461.7807421962299|[30.7377203726281...|\n",
      "| wjennings@gmail.com|6362 Wilson Mount...|      PowderBlue| 32.12538689728784|11.733861690857392|  34.8940927514398|  3.1361327164897803| 457.84769594494855|[32.1253868972878...|\n",
      "|rebecca45@hale-ba...|8982 Burton Row\n",
      "W...|       OliveDrab| 32.33889932306719|  12.0131946940144| 38.38513659413844|   2.420806160901484|  407.7045475495441|[32.3388993230671...|\n",
      "|alejandro75@hotma...|64475 Andre Club ...|            Cyan|32.187812045932155|14.715387544156501|38.244114594343515|  1.5165755808319439| 452.31567548003545|[32.1878120459321...|\n",
      "|samuel46@love-wes...|544 Alexander Hei...|   LightSeaGreen| 32.61785606282345|13.989592555825254|37.190503800397956|   4.064548550437977|   605.061038804892|[32.6178560628234...|\n",
      "+--------------------+--------------------+----------------+------------------+------------------+------------------+--------------------+-------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "output.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final_data = output.select(\"features\",'Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data,test_data = final_data.randomSplit([0.7,0.3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                374|\n",
      "|   mean| 494.77136952135584|\n",
      "| stddev|  79.32347347332397|\n",
      "|    min| 256.67058229005585|\n",
      "|    max|  765.5184619388372|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "train_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-------------------+\n",
      "|summary|Yearly Amount Spent|\n",
      "+-------+-------------------+\n",
      "|  count|                126|\n",
      "|   mean|  512.7978327643508|\n",
      "| stddev|  78.05150254141621|\n",
      "|    min|  275.9184206503857|\n",
      "|    max|  689.2356997616951|\n",
      "+-------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_data.describe().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a Linear Regression Model object\n",
    "lr = LinearRegression(labelCol='Yearly Amount Spent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Fit the model to the data and call this model lrModel\n",
    "lrModel = lr.fit(train_data,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coefficients: [25.9313842258,38.5223901033,0.061323831352,61.8542031461] Intercept: -1042.5967496966575\n"
     ]
    }
   ],
   "source": [
    "# Print the coefficients and intercept for linear regression\n",
    "print(\"Coefficients: {} Intercept: {}\".format(lrModel.coefficients,lrModel.intercept))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test_results = lrModel.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|           residuals|\n",
      "+--------------------+\n",
      "|  0.0211314098588673|\n",
      "|   5.133420657904367|\n",
      "| -22.148174773562573|\n",
      "|-0.28200824770379995|\n",
      "|  2.6981651070976795|\n",
      "|  0.7429079096369264|\n",
      "|  -3.455359493022627|\n",
      "| -2.1941889835831034|\n",
      "|   7.147573069494342|\n",
      "|   1.206688448485238|\n",
      "| -12.184920705516333|\n",
      "|  0.9395486603143581|\n",
      "|   7.893631818058566|\n",
      "| -1.6946216546640471|\n",
      "|  -9.987986847356694|\n",
      "|  -6.281630483941342|\n",
      "|  -9.000712269275198|\n",
      "|  -18.16281648837412|\n",
      "|  -9.055807893457029|\n",
      "|    3.71115734739368|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Interesting results....\n",
    "test_results.residuals.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unlabeled_data = test_data.select('features')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predictions = lrModel.transform(unlabeled_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+------------------+\n",
      "|            features|        prediction|\n",
      "+--------------------+------------------+\n",
      "|[30.5743636841713...|442.04328234820673|\n",
      "|[31.0472221394875...|  387.363978531117|\n",
      "|[31.1239743499119...|509.09522861332835|\n",
      "|[31.2606468698795...|421.60863950465523|\n",
      "|[31.3091926408918...|430.02255273283595|\n",
      "|[31.3895854806644...|  409.326703150346|\n",
      "|[31.5171218025062...|279.37378014340834|\n",
      "|[31.5761319713222...| 543.4207729729114|\n",
      "|[31.6548096756927...| 468.1158506580541|\n",
      "|[31.7366356860502...| 495.7267578070466|\n",
      "|[31.8093003166791...| 548.9568200683575|\n",
      "|[31.8293464559211...|384.21278932766063|\n",
      "|[31.8512531286083...|465.09861484873977|\n",
      "|[31.8627411090001...| 557.9927628287107|\n",
      "|[31.8648325480987...| 449.8792673241703|\n",
      "|[31.8745516945853...|398.56687473020884|\n",
      "|[31.8854062999117...| 399.1039852417507|\n",
      "|[31.9563005605233...| 565.2887482355729|\n",
      "|[32.0085045178551...| 452.2530289222125|\n",
      "|[32.0123007682454...|489.23389571856455|\n",
      "+--------------------+------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 10.744845638086671\n",
      "MSE: 115.45170778631018\n"
     ]
    }
   ],
   "source": [
    "print(\"RMSE: {}\".format(test_results.rootMeanSquaredError))\n",
    "print(\"MSE: {}\".format(test_results.meanSquaredError))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent results! Let's see how you handle some more realistically modeled data in the Consulting Project!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
